Human annotation

Text annotation

When human annotation is used, controlling for the quality of the annotation is critical. Furthermore, especially when the task is not trivial and subjective, it is also important to consider whether the annotation reflects the “true” label or idiosyncrasies of individual annotators (Geva2019are). Vincent Warmerdam‘s NormConf talk Group-by statements that save the day discusses the annotation issues with GoEmotions dataset. ^cd5cb4

Tools

• prodigy
• https://github.com/koaning/doubtlab - “Doubt your data, find bad labels” - https://www.label-sleuth.org/ - Shnarch2022Label - https://labelstud.io/
  - https://inception-project.github.io/